Skip to content

Improve token index performance by using fewer memory during search. #8152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 17, 2024
Merged

Improve token index performance by using fewer memory during search. #8152

merged 3 commits into from
Oct 17, 2024

Conversation

@isoos
Copy link
Collaborator

@isoos isoos commented Oct 17, 2024

  • Refactored initialization, as we can be more efficient if the construction of the index does not assume that entries may be removed.
  • Using int keys in the token map is slightly more efficient, but most of the gain is coming from using List<double> instead of Map.
  • The change yields about 10% of time improvements in the benchmark.
  • Added a small benchmark tool that can be used in the future for similar local checks.

@isoos isoos requested a review from jonasfj October 17, 2024 08:01
jonasfj
jonasfj reviewed Oct 17, 2024
View reviewed changes
@isoos isoos merged commit 61e2c75 into dart-lang:master Oct 17, 2024
32 checks passed
@isoos isoos deleted the token-index branch October 17, 2024 09:21
@isoos isoos mentioned this pull request Nov 4, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jonasfj jonasfj jonasfj approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@isoos @jonasfj